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Abstract 

Introduction: Practice pattern data demonstrate regional variation and lower than expected rates of adherence to 
practice guideline (PG) recommendations for the treatment of stage ll/IIIA resected and stage IIIA/IIIB unresected 
non-small cell lung cancer (NSCLC) patients in Ontario, Canada. This study sought to understand how clinical 
decisions are made for the treatment of these patients and the role of PGs. 

Methods: Surveys and key informant interviews were undertaken with clinicians and administrators. 

Results: Participants reported favorable ratings for PGs and the evidentiary bases underpinning them. The 
majority of participants agreed more patients should have received treatment and that regional variation is 
problematic. Participants estimated that up to 30% of patients are not good candidates for treatment and up 
to 20% of patients refuse treatment. The most common barrier to implementing PGs was the lack of 
organizational support by clinical administrative leadership. There was concern that the trial results 
underpinning the PG recommendations were not generalizable to the typical patients seen in clinic. The 
qualitative analysis yielded five themes related to physicians' decision making: the unique patient, the unique 
physician, the family, the clinical team, and the clinical evidence. A dynamic interplay between these factors exists. 

Conclusion: Our study demonstrates the challenges inherent in (i) the complexity of clinical decision making; (ii) how 
quality of care problems are perceived and operationalized; and (iii) the clinical appropriateness and utility of PG 
recommendations. We argue that systematic and rigorous methodologies to help decision makers mitigate or 
negotiate these challenges are warranted. 

Keywords: Practice guidelines, Clinical decision making, Practice variation, Non-small cell lung cancer, Quality of care 



Introduction 

Practice guidelines (PGs) are tools that contribute to 
changing care processes and the outcomes of care [1,2]. 
Clinical data audits and measures of practice patterns 
are methods often used to gauge the extent of adherence 
to recommendations and to identify ongoing quality is- 
sues and quality improvement successes (or failures). 
Studies looking at adherence to PG recommendations 
often show results that are interpreted as reflecting less 
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than stellar performance. As a consequence, the full po- 
tential of PGs not being achieved is the conclusion [3,4]. 

In Ontario (population approximately 13.5 million), the 
cancer system is overseen by Cancer Care Ontario (CCO), 
an advisory body to the Ontario Ministry of Health and 
Long-Term Care on all matters related to cancer. CCO 
uses its performance improvement cycle as an organizing 
quality framework [5]. The cycle is defined by four com- 
ponents: data (to measure how the system is performing), 
knowledge (evidence-based knowledge on how the system 
should perform), transfer (the transfer of this knowledge 
to various stakeholders), and performance management 
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(tools to facilitate application of knowledge). The Program 
in Evidence-based Care (PEBC) at CCO develops evidence- 
based PGs, and other evidence-resources, through a strat- 
egy of collaboration and application of rigorous methods 
[2,6]. Its contributions fit primarily in the knowledge and 
transfer quadrants of the cycle and directly inform the 
other quadrants, for example, recommendations serving as 
the foundation upon which quality indicators for the data 
quadrant are made. 

It is through an analysis of this cycle that clinical quality 
concerns were identified related to two PG documents 
targeting the clinical management of the non-small cell 
lung cancer (NSCLC) patients. Specifically, chemotherapy 
for stage II/IIIA resected NSCLC patients (PG1) and che- 
moradiation for stage IIIA/B unresected NSCLC patients 
who also had good performance status and minimal 
weight loss (PG2) are recommended [7,8]. However, an 
analysis of practice patterns data in Ontario, Canada, as 
illustrated in the 2010 to 2011 Cancer System Quality 
Index (CSQI), a yearly report card of the quality of the 
Ontario cancer system, showed diverse patterns of 
practice in relation to these lung cancer PG recommen- 
dations [9]. Specifically, 41% of resected stage II/IIIA 
patients and 25% of unresected stage IIIA/B patients 
treated at regional cancer centres (RCC) received care 
that aligned with the PG recommendations. On aver- 
age, 38% and 34% of resected stage II/IIIA Ontario pa- 
tients and unresected stage IIIA/B Ontario patients, 
respectively, were reported to have received no treat- 
ment. Finally, practice patterns varied significantly 
across the different regions in the province for both of 
these patient populations (see Tables 1 and 2). 

In response to this data, we were asked for advice on 
how to fix the problem - particularly given the high pro- 
portion of patients receiving no treatment. Thus, we 
undertook a study with the objective to engage with 
Ontario clinicians providing care to lung cancer pa- 
tients to better understand the process of clinical deci- 
sion making. Specifically, we sought answers to the 
following questions: 

1. How do clinicians interpret and perceive the 
practice pattern data? Is there agreement about 
what constitutes a potential clinical problem? 

2. How do clinicians interpret and perceive the PGs, 
specifically the evidence and the recommendations, 
which are designed to guide practice? 

3. What are the barriers and enablers clinicians 
experience in implementing the PG 
recommendations? 

4. What factors are taken into consideration as part 
of the decision making process? Where does the 
evidence and recommendations, as per the PGs, 

fit in? 



Methods 

Study context and overview 

This study took place in Ontario, Canada. As described 
above, CCO is the provincial advisor on all issues related 
to cancer. The province is divided into 14 different Local 
Health Integration Networks (LHIN) that are respon- 
sible for direct clinical management. Radiation therapy is 
provided in the regional cancer centres, systematic ther- 
apy is provided in the regional cancer centres or com- 
munity, and cancer surgery is provided in academic 
settings and hospitals throughout the province. The links 
between surgeons and the regional cancer programs in 
the province varies. 

Two PGs developed by the Lung Cancer Disease Site 
Guideline Development Group of CCO's PEBC were tar- 
geted for inquiry and are the subjects of this study [5,6]. 
(The two PGs are described in Additional files 1 and 2, 
respectively.) In brief, the first PG recommends cisplatin- 
based chemotherapy for stage II/IIIA resected NSCLC 
patients (PG1), and the second PG recommends cisplatin- 
based chemotherapy with radiation for unresected stage 
IIIA/B NSCLC patients with good performance status 
and minimal weight loss (PG2). The evidence that un- 
derpins these recommendations shows that there is a 
survival benefit for patients treated with the recom- 
mended approach. 

A mixed methods approach was used to address the re- 
search objectives. Ethics approval was received from 
McMaster University Faculty of Health Sciences/Hamilton 
Health Sciences Research Ethics Board (REB# 11-327). 
The project was funded by CCO, but the research team 
was editorially independent of the funders. 

Survey 
Participants 

Surgeons, medical oncologists, and radiation oncologists 
from Ontario were identified through multiple sources, 
including regional network leaders and network mem- 
bers in each of systemic therapy, radiation therapy, and 
surgery programs; individuals involved in lung cancer 
PG development and its application; and practicing clini- 
cians who treat lung cancer. Participants were e-mailed 
a letter of invitation outlining the study objectives and 
participation requirements; letters were cosigned, where 
appropriate, by the provincial clinical leader at CCO. 

Process 

Clinicians agreeing to participate completed the survey 
on an on-line password-protected platform. Participation 
was voluntary and confidential, and completion of the 
survey indicated consent to participate. Reminder e- 
mails were sent out on weeks two, four and eight after 
the initial invitation was distributed. 
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Table 1 Percentage of resected NSCLC stage II and MIA patients treated with practice guideline recommended adjuvant 
chemotherapy following surgery by Local Health Integration Network (LHIN) Region of Residence (patients diagnosed 
in 2009) 



LHIN 


Guideline treatment (RCC) 


Treated non-RCC 


Alternate treatment 


No treatment <120 days of Surgery 


Total resected 
ll&IIIA cases 


1 


32% 


6% 


3% 


59% 


34 


2 


38% 


4% 


4% 


53% 


68 


3 


16% 


19% 


19% 


47% 


43 


4 


46% 


2% 


7% 


46% 


61 


5 


10% 


41% 


10% 


38% 


29 


6 


32% 


27% 


5% 


35% 


37 


/ 


29% 


32% 


5% 


34% 


65 


8 


40% 


25% 


3% 


32% 


68 


9 


49% 


11% 


6% 


34% 


83 


10 


52% 


4% 


11% 


33% 


27 


1 1 


61% 


1% 


7% 


31% 


72 


12 


37% 


24% 


11% 


28% 


46 


13 


50% 


4% 


4% 


42% 


50 


14 


65% 


0% 


10% 


25% 


20 


Ontario 


41% 


14% 


7% 


38% 


703 



NSCLC, non-small cell lung cancer; RCC, Regional Cancer Centre. 

Report date: December, 201 1. 

Data source: Cancer Care Ontario, ALR, OCR. 

Notes: 

Many patients in the "No Treatment" category may not be medically fit for the practice guideline treatment due to factors we are not currently able to adjust for. 
Patients may also decline treatment for personal reasons. Others may have been treated outside Ontario. 

Alternate Treatment: cases receiving a therapy different from that recommended in the practice guidelines. This may include non-platinum-based chemotherapy 
or radiation therapy only. 

Treated Non RCC: patients receiving chemotherapy outside of a cancer centre where the drug regimen is not reported to Cancer Care Ontario (CCO). 
Parts of this material are based on data and information compiled and provided by the Canadian Institute for Health Information. However, the analyses, 
conclusions, opinions and statements expressed herein are those of the author, and not necessarily those of the Canadian Institute for Health Information. 



Survey 

The survey consisted of the following areas of inquiry: 

1. Assessment of the PG Recommendations and the 
Evidentiary Base. Participants reviewed the abridged 
version of the PG and completed a survey relating to 
the recommendations (participants' awareness and 
agreement with the PG recommendations, 
perceptions about their quality and 
implementability, and assessment of alignment with 
his/her clinical practice) and the evidentiary base 
underpinning the PG recommendations 
(completeness, validity). Each of these items was 
answered using a 7-point response scale (1 = totally 
disagree to 7 = totally agree). 

2. Perceptions of Practice Patterns. For each PG, the 
overall provincial and regional practice pattern data 
from the CSQI were presented to the participant and 
summarized proportion of patients receiving no 
treatment, treatment adherent to PG 
recommendations, alternative treatment, and 
treatment outside of the cancer centre. The participant 
was asked to what extent the patterns aligned with his/ 



her expectations and if they were clinically appropriate 
or indicative of a quality of care problem. 

3. Benchmarks. Based on his/her own clinical 
experience, the participant was asked to estimate (a) 
the proportion of patients (in both populations) who 
refuse treatment even though they are appropriate 
candidates and (b) the proportion of resected stage 
II/IIIA patients who are not appropriate candidates 
(due to poor performance status, comorbidities) for 
cisplatin-based chemotherapy. 

4. Barrier Analysis. Participants were presented with a 
series of barriers and enablers to the adoption of PG 
recommendations and asked to indicate which ones 
they perceived to be true. The questions were 
informed by the BARRIERS scale [10,11]. 

5. Demographics. Participant demographics regarding 
age, discipline, setting of practice, and location of 
practice were collected. 

Data analysis 

Descriptive and frequency analyses were undertaken and 
all members of the team participated in the interpret- 
ation of the results. 
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Table 2 Percentage of unresected NSCLC stage IMA and 1MB patients treated with practice guideline recommended 
chemo-radiation therapy by Local Health Integration Network (LHIN) Region of Residence (patients diagnosed in 2009) 



LHIN 


Guideline treatment 
(RCC) 


Treated non- 
RCC 


Alternate 
treatment 


No treatment < 180 days of 
diagnosis 


Total nonresected MIA & 1MB 
cases 


1 


36% 


6% 


27% 


31% 


84 


2 


35% 


6% 


24% 


34% 


140 


3 


25% 


13% 


34% 


28% 


53 


4 


25% 


1% 


41% 


33% 


181 


5 


6% 


6% 


39% 


48% 


31 


6 


21% 


11% 


32% 


36% 


85 


/ 


10% 


16% 


35% 


38% 


79 


8 


12% 


15% 


27% 


46% 


95 


9 


21% 


12% 


25% 


41% 


138 


10 


34% 


0% 


35% 


31% 


68 


1 1 


32% 


2% 


41% 


25% 


128 


12 


19% 


11% 


38% 


32% 


47 


13 


28% 


6% 


46% 


20% 


69 


14 


18% 


3% 


50% 


29% 


38 


Ontario 


25% 


7% 


34% 


34% 


1,236 



NSCLC, non-small cell lung cancer; RCC, Regional Cancer Centre. 

Report date: November, 201 1. 

Data source: Cancer Care Ontario, ALR, OCR. 

Notes: 

"No Treatment" does not necessarily indicate inappropriate care. Many patients may not be medically fit for the practice guideline treatment due to factors we are 
not currently able to measure. Some patients may also decline treatment for a variety of personal reasons. Others may have been treated outside Ontario. 
Alternate Treatment: cases receiving a therapy different from that recommended in the practice guidelines. This may include non-platinum-based chemotherapy 
or chemotherapy only or radiation therapy only. 

Treated Non-RCC: patients receiving chemotherapy outside of a cancer centre where the drug regimen is not reported to Cancer Care Ontario (CCO). 
Parts of this material are based on data and information compiled and provided by the Canadian Institute for Health Information. However, the analyses, 
conclusions, opinions and statements expressed herein are those of the author, and not necessarily those of the Canadian Institute for Health Information. 



Key informant interviews 
Overview 

A grounded theory approach was used for the key in- 
formant interviews [12]. Two specific questions were 
used to direct the inquiry: 

1. How do Ontario physicians construct treatment 
decisions regarding care for their resected stage II/ 
IIIA NSCLC patients and their unresected stage 
IIIA/B NSCLC patients? 

2. What is the role of the knowledge products, such 
as the PEBC PGs, in the decision making process? 

Participants and process 

Following methodological standards for qualitative inquiry 
[13], purposive sampling was undertaken using clinician 
discipline, region and setting as the criteria for selection. 
Sampling continued until theoretical saturation was 
reached (i.e., new data generated no new insights into the 
findings). Letters of invitation were sent to a selected sam- 
ple of candidate participants. In-person or telephone inter- 
views were scheduled for those who agreed to participate. 
The semi-structured interview guide was pilot-tested and 



revised, with the final version used to guide the interview 
(see Additional file 3). 

Data analysis 

As per grounded theory methods [12], data analysis and 
data collection (including participant interviews, field 
notes, and analytic memos) proceeded iteratively, with 
the preliminary findings informing directions for subse- 
quent interviews. Interviews were transcribed and coded 
following conventional grounded theory procedures and 
theory-building stages [14]. 

Results 

Survey 
Participants 

A total of 71 responses were received (response rate 30%). 
The majority of participants were men; surgeons; aged be- 
tween 35 and 60 years; from the Hamilton Niagara Haldi- 
mand Brant region; and working in a cancer centre. 

Assessment of recommendations and evidentiary base 

For PG1 (resected stage II/IIIA) and PG2 (unresected 
stage IIIA/B), the specific recommendations were rated 
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very favorably by the participants (see Table 3). Indeed, 
the response modes for each item for both PGs tended 
to be at the extreme ends of the scale. Also, there were 
favorable assessments of the evidentiary bases underpin- 
ning both PGs (see Table 4), with a slight preference for 
the evidentiary base for PG1. The differences in quality 
ratings between PG1 and PG2 were greater for the as- 
sessments of the evidentiary bases than for the assess- 
ments of the recommendations. 

Perceptions of practice patterns 

As it relates to PG1 (for resected stage II/IIIA patients), 
61% of the participants reported that the percentage of 
patients receiving treatment according to recommenda- 
tions was less than what they expected, 3% reported it 
was more than what they expected, and 36% reported 
that it was as they expected. Further, while 40% of re- 
spondents reported the percentage of patients receiving 
no treatment was too low to be clinically appropriate, 
3% reported it was too high to be clinically appropriate, 
and 46% reported it was about right and clinically appro- 
priate. When asked if the variation in the regional patterns 
of practice across the province was an indication of a 



quality of care problem, 30% of respondents disagreed, 
48% of respondents agreed, and 23% of respondents 
were neutral. 

With respect to PG2 (for unresected stage IIIA/B pa- 
tients), 54% of participants reported that the percentage 
of patients receiving PG recommended treatment was 
less than they expected, 5% reported it was more than 
they expected, and 41% reported it was as they expected. 
Further, only 27% of respondents reported the percent- 
age of patients receiving no treatment was too low to be 
clinically appropriate, 22% thought it was too high to be 
clinically appropriate, and 51% thought it was about 
right and clinically appropriate. When asked if the vari- 
ation in the regional patterns of practice across the prov- 
ince was an indication of a quality of care problem, 36% 
of respondents disagreed, 37% of respondents agreed, 
and 26% of respondents were neutral. 

Estimating benchmarks 

The range in participants' estimates of resected stage II/ 
IIIA patients not eligible for treatment because of poor 
performance status or comorbidities was large, but over 
half estimated that between 15% and 30% of patients 



Table 3 Practice guideline 1: resected NSCLC stage II and IIIA patients 



Item 


Rating percent A 
Disagree Neutral 


Agree 


Mean 


Sd 


Mode 


Assessment of the Recommendations 














a. Aware of the recommendations 


4.9 


2.4 


92.7 


6.3 


1.2 


7.0 


b. Agree with the recommendations 


5.0 


0 


95.0 


6.3 


1.1 


7.0 


c. Recommendations are unambiguous 


7.5 


7.5 


85.0 


5.9 


1.5 


7.0 


d. Recommendations are supported by the evidence 


2.5 


0 


97.5 


6.4 


0.9 


7.0 


e. Recommendations are current 


/./ 


2.6 


89.7 


6.0 


1.3 


6.0 


f. Recommendations are easy to apply in their clinical context 


2.6 


10.3 


87.2 


6.2 


1.3 


7.0 


g. Recommendations are too rigid for the patients they are intended 


68.4 


10.5 


21.1 


2.9 


1.8 


2.0 


h. Recommendations do not align with how they typically manage these patients 


80.0 


5.0 


15.0 


2.4 


1.8 


1.0 a 


i. Recommendations apply to the patients they target 


7.9 


0 


92.1 


6.0 


1.3 


6.0 


j. Recommendations are biased 


82.9 


9.8 


7.3 


2.0 


1.6 


1.0 


k. Support for the recommendations 


5.0 


2.5 


92.5 


6.3 


1.1 


7.0 


1. Clinical practice of respondent aligns with the recommendations 


5.0 


5.0 


90.0 


6.2 


1.1 


7.0 


Assessment of the Evidentiary Base 














a. It is complete 


/./ 


5.1 


87.2 


6.0 


1.2 


7.0 


b. It is convincing 


2.6 


5.1 


92.3 


6.1 


1.0 


7.0 


c. It is informative 


7.5 


0 


92.5 


6.1 


1.2 


7.0 


d. It is relevant to typical patients 


12.5 


2.5 


85.0 


6.0 


1.6 


7.0 


e. It is strong 


5.3 


2.6 


92.1 


5.9 


1.2 


6.0 


f. It is current 


10.5 


2.6 


86.8 


5.8 


1.5 


6.0 a 



NSCLC, non-small cell lung cancer. 

A Disagree (ratings 1 to 3 on 7-point scale); Neutral (rating 4 on 7-point scale}; Agree (rating 5 to 7 on 7-point scale). 
a Multiple mode exists. The smallest value is shown. 
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Table 4 Practice guideline 2: unresected NSCLC stage MIA and 1MB Patients 



Item 


Rating percentage* 
Disagree Neutral 


Agree 


Mean 


SD 


Mode 


Assessment of the Recommendations 














a. Aware of the recommendations 


10.0 


2.5 


87.5 


6.0 


1.6 


7.0 


b. Agree with the recommendations 


5.4 


5.4 


89.2 


6.0 


1.2 


7.0 


c. Recommendations are unambiguous 


5.3 


7.9 


86.8 


5.8 


1.2 


6.0 


d. Recommendations are supported by the evidence 


5.3 


2.6 


92.1 


5.9 


1.0 


6.0 


e. Recommendations are current 


2.6 


10.5 


86.8 


5.8 


1.2 


6.0 


f. Recommendations are easy to apply in their clinical context 


5.3 


7.9 


86.8 


5.8 


1.3 


6.0 


g. Recommendations are too rigid for the patients they are intended 


76.3 


5.3 


18.4 


2.8 


1./ 


2.0 


h. Recommendations do not align with how they typically manage these patients 81.6 


7.9 


10.5 


2.3 


1.5 


2.0 


i. Recommendations apply to the patients they target 


2.6 


5.3 


92.1 


5.9 


1.0 


6.0 


j. Recommendations are biased 


89.7 


10.3 


0 


1.8 


0.9 


2.0 


k. Support for the recommendations 


2.6 


/./ 


89.7 


6.1 


1.0 


6.0 a 


I. Clinical practice of respondent aligns with the recommendations 


2.6 


/./ 


89.7 


6.1 


1.0 


6.0 


Assessment of the Evidentiary Base 














a. It is complete 


5.3 


13.2 


81.6 


5.7 


1.3 


6.0 


b. It is convincing 


2.6 


10.5 


86.8 


5.7 


1.2 


5.0 a 


c. It is informative 


2.6 


7.9 


89.5 


5.9 


1.2 


7.0 


d. It is relevant to typical patients 


2.6 


2.6 


94.7 


5.9 


1.1 


7.0 


e. It is strong 


5.4 


13.5 


81.1 


5.5 


1.3 


6.0 


f. It is current 


5.3 


18.4 


76.3 


5.7 


1.4 


7.0 



NSCLC, non-small cell lung cancer. 

a Multiple mode exists. The smallest value is shown. 

A Disagree (ratings 1 to 3 on 7-point scale); Neutral (rating 4 on 7-point scale); Agree (rating 5 to 7 on 7-point scale). 



would not be appropriate. Participants' estimates of the 
proportion of resected stage II/IIIA patients who refuse 
treatment although they are appropriate candidates was 
again large, but almost two-thirds estimated that be- 
tween 15% and 30% of their patients meet these criteria. 
Finally, over 90% of participants' estimated that between 
5% and 20% of unresected patient stage IIIA/B patients 
refuse treatment even when they are appropriate candi- 
dates for this treatment. 

Barrier analysis 

The most commonly reported barriers to implementing 
the PG recommendations were lack of organizational 
support by clinical administrative leadership (e.g., clinical 
head of the medical oncology program) (31.7% agree- 
ment); slow referral process (30.8% agreement); beliefs 
that patients seen in clinic do not reflect the patients in- 
cluded in the studies comprising the evidentiary base 
(25% agreement); and perceptions that it is not easy for 
patients in the region to gain access to the recom- 
mended treatment (23.1% agreement). In contrast, an 
unreliable referral process to the cancer centre/specialist 
(10% agreement); inadequate medical expertise in the re- 
gion (9.8% agreement); belief that PG recommendations 



are not cost effective (7.5% agreement); lack of clinical 
skill of respondent (4.9% agreement); and perceptions 
that adverse effects of treatment were unacceptable 
(2.7% agreement) were the barriers that garnered the 
least support by participants. See Table 5 for full barrier 
analysis results. 

Key informant interviews 

In total, 21 participants (6 medical oncologists, 4 sur- 
geons/surgical oncologists, 6 radiation oncologists, and 5 
healthcare administrators) were interviewed; similar 
findings emerged from interviews with both clinicians 
and administrators. Five theoretical categories emerged 
concerning: (i) the unique patient, (ii) the family, (iii) the 
physician, (iv) the clinical team, and (v) the clinical evi- 
dence. We found a dynamic interplay between these fac- 
tors in both clinical encounters and the construction of 
treatment decisions. 

The unique patient 

The unique patient theme was comprised of three factors, 
including specific clinical circumstances; demographics; and 
social and personal contexts. Age was mentioned frequently 
as a defining feature that determined appropriateness of the 
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Table 5 Barrier analysis 



Item 




Rating 




Mean 


Sd 


Mode 




Disagree 


Neutral 


Agree 








a. Surgeons are reluctant to refer patients to a medical onco ogist and/or radiation oncologist. 


oj.U 


Z.J 


1 Z.J 


1 Q 


1 A 


1 ,U 


b. The referral process to a cancer centre or cancer specialist is complex. 


76.2 


11.9 


11.9 


2.5 


1.6 


1.0 


c. The referral process to a cancer centre or cancer specialist is slow. 


53.8 


15.4 


30.8 


3.6 


1.8 


3.0 


d. The referral process to a cancer centre or cancer specialist is unreliable. 


82.5 


7.5 


10.0 


2.3 


1.5 


1.0 


e. Personal lack of clinical skill to implement the recommendations. 


90.2 


4.9 


4.9 


1.6 


1.3 


1.0 


f. Organizational support from the clinical administrator leaders exists in the institution to support 
the implementation of the recommendations. 


31.7 


9.8 


58.5 


4.5 


2.3 


6.0 a 


g. There is adequate medical expertise in their region to implement the recommendations. 


9.3 


4./ 


86.0 


6.1 


1.6 


7.0 


I — i— 1 __■__■ __l __ 1 ■ ■■ .1 ■ 1 __■ 1 1 ri.i_.i_l __ ■ 1 __■____! 

h. The patients in the studies comprising the evidentiary base do not reflect the typical patient they 
seen in the clinic. 


60.0 


15.0 


25.0 


3.1 


1.8 


2.0 


i. It is easy for patients in their region to access the recommended treatment 


23.1 


5.1 


71.8 


5.1 


1.6 


5.0 a 


j. The implementation of the recommendations will result in unacceptable levels of adverse effects 
for the typical resected stage II and IIIA NSCLC patients seen in practice. 


86.5 


10.8 


2.7 


2.1 


1.1 


2.0 


k. Optimizing the treatment of lung cancer patients is not as much of an organizational priority in 
their care setting as is the treatment of patients with other cancer diagnoses. 


66.7 


17.9 


15.4 


2.6 


1.7 


1.0 


1. The implementation of the recommendations will yield the anticipated benefits as per the 
recommendations/guideline. 


66.7 


17.9 


15.4 


2.8 


1.6 


2.0 


m. The recommendations are not cost effective. 


77.5 


15.0 


7.5 


2.4 


1.3 


1.0 



' Multiple mode exists. The smallest value is shown. 



treatment options, particularly for the unresected stage 
IIIA/IIIB patients, where trade-offs between benefit and risk 
are much smaller than for resected stage II/IIIA patients. 
Also important in this theme were the specific personal and 
social circumstances of the patients and their wishes with 
regard to the decisions made. For example, patients living 
alone or who had no access for transportation were consid- 
ered less favorable candidates for chemotherapy because of 
the demands of the regimens, propensity of adverse effects, 
and risks for toxicities that may be difficult to manage alone 
or without easy access to a care provider. 



or not). Family members emerged as key influentials in 
determining what treatment options would be encouraged 
or discouraged. Attitudes and perceptions of recom- 
mended treatments were influenced by family members 
who had gone through a similar treatment or serious ill- 
ness. For example, a positive experience yielding an 'I am 
a survivor' narrative would influence how they communi- 
cated options to their 'patient' family member. Family 
members also served as a larger window into the patient's 
life and helped the physicians understand the patient's 
daily functioning and what the patient's lifestyle was like. 



The unique physician 

This theme was defined by physicians' experiences, 
knowledge and judgments; physician recommendations; 
and the decision making style. Of particular note was 
the fact that personal clinical styles of the physician (e.g., 
those favouring more aggressive or less aggressive treat- 
ment approaches) and experiences from previous pa- 
tients (e.g., degree of benefit or severity of adverse 
effects previously observed) were important factors that 
influenced how care options were presented and what 
were the preferred PG recommendations of the individ- 
ual physician. 

The family 

The family played a significant role in the decision mak- 
ing process. Presence or absence of family support and 
an accessible social network influenced the options offered 
to the patients and the intent of care (e.g., palliative intent 



The clinical team 

The treatment decision making process for NSCLC pa- 
tients is not an isolated event between the physician and 
patient, but instead is a dynamic process that involves 
engagement of specialists and colleagues. Participants in- 
dicated that this can be more or less formal (informal 
chats or multidisciplinary cancer conferences), but it is a 
valued element in the decision making process. This is 
particularly true of the unresected stage IIIA and IIIB 
patients, whose care needs are much more complex and 
who present in a much more heterogeneous fashion. 

The clinical evidence 

There was a tendency for each physician to apply his/ 
her interpretation of the medical literature and to find 
and extrapolate new data to apply to his/her individual 
patient's case. This could be done implicitly or explicitly. 
The predominant treatment styles or practices (e.g., 
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aggressive versus conservative) at an institutional- or 
regional-level also had a strong influence in how evi- 
dence was scrutinized, with normative influences (e.g., 
'this is how we do it here') playing a very influential 
role, regardless of whether there was alignment with 
the evidence. 

The CCO PEBC PGs were seen as key foundational 
tools to the decision making process. However, while the 
CCO Lung PGs do identify the characteristics of the pa- 
tient group most suitable for treatment, these important 
caveats are defined by the patient eligibility criteria in 
the clinical trials comprising the evidentiary base. In- 
deed, the participants expressed concern that PGs typic- 
ally tackle issues and provide guidance only on those 
topics for which there exists primary studies to inform 
practice. In contrast, the diversity of the actual patient 
population and the contexts in which care is provided 
are much greater than the circumstances that the pri- 
mary studies capture. Thus, for example, the PG recom- 
mendations related to care for unresected stage IIIA/IIIB 
NSCLC are quite limiting given that the patients who 
present in real life are much more variable with respect 
to functional status and comorbidities than the stage 
IIIA/ IIIB patients reflected in the evidence and PG rec- 
ommendations. This lack of connection between the real 
life patient and the study patient can undermine the 
value, relevance and utility of the PG. 

Discussion 

When approached to undertake this project, our team was 
presented with practice pattern data assumed to be re- 
flective of a quality of care problem. Of particular concern 
was that a significant proportion of patients received no 
treatment, only a modest percentage of patients received 
treatment that aligned with the PG recommendations, 
and considerable regional variation existed within each of 
the clinical care options. We believe this project speaks to 
several issues relevant to the practice and scholarship of 
knowledge translation: (i) defining whether a quality of 
care problem exists or not is more complex than consider- 
ation of practice pattern data alone; (ii) limitations to an 
evidence base and the recommendations it informs in a 
PG compound this problem; and (iii) systematic and rigor- 
ous methodologies to help decision makers mitigate or ne- 
gotiate these challenges are warranted. These issues will 
be addressed below. 

Not surprisingly, and articulated by participants, there 
are limitations with clinical databases on which the prac- 
tice patterns are based. These limitations related to the 
completeness, validity and reliability of the data [15,16]. 
The assessment of cancer PG adherence, particularly in 
the case of NSCLC, requires a complete and accurate 
capture of a patient's stage, functional status, and weight 
loss; these factors directly influence the selection of 



optimal treatment and determining which patients are 
the best candidates for recommended therapy. Although 
this information is obtained during the clinical encoun- 
ter, it may or may not be captured routinely in electronic 
databases. Similarly, data regarding the use and results 
of multidisciplinary case conferences (MCC) are rou- 
tinely not captured. MCCs, the practice of bringing spe- 
cific patient cases (often complex cases, as would be 
expected with lung cancer treatment) and care plans to 
a multidisciplinary team for feedback, is a key strategy to 
help determine the most optimal, safe, and preferred 
therapy tailored to the individual [17,18]. MCCs were 
formally introduced in Ontario at a system level in 2008. 
How they are specifically operationalized and function 
varies according to LHIN and specific disease sites. 

Together, the limitations with patient-specific out- 
comes and MCC processes, make fair and valid interpre- 
tations of practice pattern data more problematic. For 
example, while 30% of patients categorized as 'no treat- 
ment' is large, the practice pattern data do not differenti- 
ate between appropriate reasons for no treatment (e.g., 
poor functional status, significant weight loss, patient re- 
fusal) from inappropriate reasons for no treatment (e.g., 
patients not being offered treatment, undue influence of 
patient families). From the survey data, participants were 
most likely to estimate that between 15% and 30% of 
Stage II/IIIA patients are not appropriate candidates for 
recommended treatment and that between 5% to 20% of 
patients refuse treatment. While these data are subject 
to recall bias, we have no reliable province-wide infor- 
mation to provide reliable estimates of 'no treatment' 
patients who were simply not medically fit for treat- 
ment or the extent to which a multidisciplinary team 
came to a consensus on that plan of action. Improve- 
ments in electronic medical record technology and 
application of other, albeit expensive and time-intensive, 
methodologies (e.g., chart review) may help address some 
of these problems. 

Practice pattern data challenges are compounded by 
limitations to the PGs themselves. Similar to what has 
been found elsewhere [19], two thirds of our survey par- 
ticipants disagreed that the application of the PG recom- 
mendations would yield the anticipated benefits as seen 
in the primary studies, and our key informants reported 
that the patients included in the clinical trials on which 
the PGs were based do not reflect the typical patients 
seen in a cancer centre. For the clinicians, their patients 
are commonly more elderly, frail, and have comorbidi- 
ties that make administration of the PG recommended 
treatment difficult. Thus, some degree of practice vari- 
ation would and should be expected. These data raised a 
seemingly important paradox between perceptions of PG 
quality, the perceptions of the clinical acceptability or 
utility of PG recommendations, and their purpose. 
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As was found in the original external review of the tar- 
geted PGs as part of its development process [7,8], survey 
participants gave favorable ratings to both the evidence 
base (i.e., the systematic review) and the recommenda- 
tions. Similarly, the key informants reported that the 
PEBC PGs are an important component of the decision 
making process. Thus, while data show that PGs may be 
methodologically sound, credible, and of high quality, the 
clinical relevance or clinical validity of the recommenda- 
tions may not be optimal. Indeed, current standards for 
PG development, reporting or appraisal (e.g., AGREE II 
[20], Institute of Medicine Standards [21]) target the entire 
guideline process. 

More recent efforts have begun to focus directly on the 
interplay between evidence, its context, the system, and 
the recommendations. For example, several international 
collaborative research efforts (e.g., AGREE-REX [www. 
agreetrust.org], DECIDE [www.decide-collaboration.eu]) 
are interested in advancing the science and practice of 
PGs by creating resources, tools and platforms targeted 
specifically to the recommendations (vs. the entire PG de- 
velopment process) and optimizing their implementability. 
Others have focused tactics to inform at the policy level 
[22]. Efforts to accelerate and better coordinate these 
efforts are warranted. More evaluation is required to 
determine tactics and strategies that are more and less 
effective. Further, exploring innovative statistical methods 
to facilitate systematic reinterpretation of the evidence from 
a lens of its probabilistic limitations would also be an asset. 
This may be of particular importance at the policy and sys- 
tem levels, where decision makers may be less familiar with 
evidence and data, their strengths and limitations, and 
where, in the absence of alternatives, PGs may be viewed as 
something more than a tool to guide clinical practice. 

The qualitative analysis demonstrates that while PGs 
and data are important, they represent only one compo- 
nent of a complex decision making process. In a health- 
care system, quality often focuses on adherence to 
evidence (consider the CCO quality improvement cycle 
[5]). This is not surprising given we likely have the most 
robust, albeit still very incomplete, methods to do this. 
However, less understood is how to fit the individual pa- 
tient, the unique clinicians, the family, the clinical team, 
and the evidence - together - into a quality sphere or 
metric. In an environment in which patient-centred care 
is becoming an increasingly important and prioritized 
goal, understanding what this means and how to achieve 
this will be important [23]. 

In our analysis, we believe that the strongest signal of 
a quality of care problem is not the overall provincial 
pattern data (though, here too, we are concerned) 
but rather the significant variation between regions 
in the province. While there are important normative 
differences (e.g., regions that are more or less aggressive 



in their treatment approaches), patient population differ- 
ences (e.g., socioeconomic class linked to cancer risk 
and health-seeking behavior), and familial system differ- 
ences (e.g., how cancer is perceived, discussed) that may 
account for some of the variability, can it account for 
the magnitude reported? The barrier analysis found, as 
has been seen elsewhere [19], that referral processes to 
the cancer centres or oncology specialists were perceived 
as slow (30.8%) and not always easy (23.1%). However, 
again, the data are able to take this analysis so far. We are 
unable to systematically and reliably deconstruct these 
barriers to a regional level. For example, we do not have a 
systematic strategy to differentiate between patients who 
were or were not referred to a cancer specialist following 
surgery, and we cannot identify individual clinicians or 
clinician teams who may favour one treatment stream 
over another. Yet, each of these components is important 
to the interpretation of the data or the decision making 
process, and these issues are important in determining 
next steps. The strategies to tackle a regional variation 
problem may be very different from strategies to tackle a 
PG limitation problem; strategies for action need to be 
carefully tailored to the problem to optimize success. Fur- 
ther research to help differentiate the relative contribu- 
tions of different challenges would be welcome. 

We believe this study highlights important issues to the 
practice and scholarship of knowledge translation as a 
whole. First, from a practice perspective, the limitations of 
practice pattern data may compromise the ability to fairly 
prioritize and choose from the range of quality of care 
problems that clinicians and policy makers can tackle, par- 
ticularly in an environment of limited resources and where 
healthcare equity is valued. This problem can be com- 
pounded in environments where quality and completeness 
of data varies across disease conditions, creating an 'un- 
fair' playing field and pitting one health condition 
against another. Similarly, a consequence of question- 
able external validity of PG recommendations may be 
the inadvertent and inappropriate prominence given to 
some care options over others. Decision makers, clini- 
cians and patients may expect more dramatic outcomes 
based on recommendations. 

Data limitations and PG limitations also present chal- 
lenges to advancing knowledge translation as a scientific 
field. The field wants to contribute to the generation of 
new knowledge that will ultimately yield a measureable 
positive impact on patients and populations. Within the 
context of PGs, this may mean, for example, the introduc- 
tion of a new technology (e.g., a PG recommending 
chemotherapy), examining its impact on processes of care 
(e.g., administration of chemotherapy to appropriate and 
eligible patients), and outcomes (e.g., increased survival). If 
the outcomes do not align with hypotheses or expectations 
(i.e., practice patterns are not as anticipated or survival 
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rates do not increase), it is difficult to attribute the causal 
meaning to the findings (e.g., problems with the PG, prob- 
lems with measures, etc.). Moreover, and importantly, a 
PG is not simply a product that provides recommenda- 
tions. It is a social and scientific process that builds a cul- 
ture of stakeholders (historically clinicians and now more 
so patients and decision makers) that are receptive to, 
understand strengths and limitations of, and are able to 
use evidence as part of their decisions [2,6]. Thus, under- 
standing what mechanism in a PG is the 'change agent' is 
difficult to ascertain. 

We would argue that disclosing the limitations of prac- 
tice pattern data and PGs are not sufficient. This is done, 
and this is done often fairly well. What are required are 
credible and reasonable methods to understand how to 
account and navigate the limitations and the appropriate 
modifications to make in response. While progress has 
been made, a more coordinated plan of action to avoid 
duplication and maximize the limited resources to ad- 
vance this goal is warranted. The scientific community 
needs to go beyond describing components or themes of 
contextualization (although this is an important first step) 
to specific action steps to address them. While avoiding 
an environment in which 'the good becomes the enemy of 
perfect' is important, the research community must not be 
complacent. Even in the best circumstances, there is the 
risk that PGs can be so routinized that the individual core 
messages from each report are lost. In response, derivative 
knowledge tools that reflect the key PG messages (for cli- 
nicians and for administrators) may be useful - reminders, 
use of champions or key influentials, problem-based learn- 
ing, and other knowledge 'boosters' - to juxtapose the PG 
recommendations against clinical data to promote discus- 
sion, debate and change [24] . 

While one of the strengths of any PG is that it is linked 
to the evidence, paradoxically, this can also present a chal- 
lenge, as discussed. The evidence from carefully designed 
phase III randomized controlled trials cannot account for 
the range of important comorbid conditions that may be 
commonly present at a clinic or the range of other factors 
such as the individuality of a given patient, the impact of 
family, and social or institutional contexts that may signifi- 
cantly influence what care option is offered or advised. 
Recognition of these factors, methods to facilitate a more 
transparent use of expert knowledge and consensus of 
opinion where this knowledge may best be found, and the 
greater use of pragmatic trials by primary researchers and 
subsequently by PG developers [25], may lead to a more 
fulsome discourse and more implementable recommenda- 
tions. Benchmarks of adherence that adopt estimates and 
ranges of what constitutes reasonable variation would be 
advantageous to better contextualize clinical practice pat- 
tern data and to more accurately ascertain the risk that a 
quality problem exists. 



A strength of our study is that we designed and ap- 
plied high quality qualitative and quantitative methods 
to understand the problem. While the clinical context of 
this study targeted cancer and the jurisdictional context 
targeted Ontario, Canada, we believe that the issues 
raised are generalizable across disease types and jurisdic- 
tions. Many health authorities and governing health bod- 
ies (e.g., Unites States, the United Kingdom) rely on 
practice pattern data to determine healthcare priorities, 
monitor progress toward quality improvement goals, 
and direct investments and resources. All PG develop- 
ment groups confront limitations of the evidence base 
and challenges with generalizing evidence to the range 
of patients faced in the healthcare system. 

A significant limitation of our study is that although 
appropriate methods were used to optimize response 
rates (co-signatures, repeated reminders, etc.), a disap- 
pointing response rate emerged with the quantitative 
component (30%), leading to the risk of a biased partici- 
pant sample. However, for most of the surveys, the 
confidence intervals were small, and for most items, par- 
ticipants focused on the extremes of the scale. While this 
explanation should not mitigate all concerns about se- 
lection bias and low response rates, perhaps, as these 
data show, the absolute variability in perceptions is lim- 
ited as well. A second limitation is the threat of recall 
bias, particularly in participants' judgments about the es- 
timated proportion of patients for whom treatment is 
not appropriate or who refuse treatment. 

Conclusion 

This study used a mixed methods approach to better 
understand how clinical problems are defined and the 
role of PGs in addressing clinical problems and inform- 
ing clinical decisions. Our study demonstrates the chal- 
lenges inherent in (i) the complexity of clinical decision 
making, (ii) how quality of care problems are perceived 
and operationalized, and (iii) the clinical appropriateness 
and utility of PG recommendations. The PG community 
(i.e., developers, users, researchers) needs to accelerate 
the development of methods and strategies to make 
practice pattern data interpretation, PGs, and PG recom- 
mendations more clinically valid, relevant and useful. 
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